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Abstract 



This paper describes a visual object detection framework that is capable of pro- 
cessing images extremely rapidly while achieving high detection rates. There are three 
key contributions. The first is the introduction of a new image representation called the 
"Integral Image" which allows the features used by our detector to be computed very 
quickly. The second is a learning algorithm, based on AdaBoost, which selects a small 
number of critical visual features and yields extremely efficient classifiers [4]. The 
third contribution is a method for combining classifiers in a "cascade" which allows 
background regions of the image to be quickly discarded while spending more com- 
putation on promising object-Uke regions. A set of experiments in the domain of face 
detection are presented. The system yields face detection performance comparable to 
the best previous systems [16, 11, 14, 10, IJ. Implemented on a conventional desktop, 
face detection proceeds at 15 frames per second. 
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1 Introduction 

This paper brings together new algorithms and insights to construct a framework for 
robust and extremely rapid object detection. This framework is demonstrated on, and 
in part motivated by, the task of face detection. Toward this end we have constructed 
a frontal face detection system which achieves detection and false positive rates which 
are equivalent to the best published results [16, 11, 14, 10, 1]. This face detection 
system is most clearly distinguished from previous approaches in its ability to detect 
faces extremely rapidly. Operating on 384 by 288 pixel images, faces are detected 
at 15 frames per second on a conventional 700 MHz Intel Pentium 111. In other face 
detection systems, auxiliary information, such as image differences in video sequences, 
or pixel color in color images, have been used to achieve high frame rates. Our system 
achieves high frame rates working only with the information present in a single grey 
scale image. These alternative sources of information can also be integrated with our 
system to achieve even higher frame rates. 

There are three main contributions of our object detection framework. We will in- 
troduce each of these ideas briefly below and then describe them in detail in subsequent 
sections. 

The first contribution of this paper is a new image representation called an integral 
image that allows for very fast feature evaluation. Motivated in part by the work of 
Papageorgiou et al. our detection system does not work directly with image intensities 
[8]. Like these authors we use a set of features which are reminiscent of Haar Basis 
functions (though we will also use related filters which are more complex than Haar 
filters). In order to compute these features very rapidly at many scales we introduce 
the integral image representation for images (the integral image is very similar to the 
summed area table used in computer graphics [2] for texture mapping). The integral 
image can be computed from an image using a few operations per pixel. Once com- 
puted, any one of these Harr-like features can be computed at any scale or location in 
constant time. 

The second contribution of this paper is a method for constructing a classifier by 
selecting a small number of important features using AdaBoost [4]. Within any im- 
age sub-window the total number of Harr-like features is very large, far larger than 
the number of pixels. In order to ensure fast classification, the learning process must 
exclude a large majority of the available features, and focus on a small set of critical 
features. Motivated by the work of Tieu and Viola, feature selection is achieved through 
a simple modification of the AdaBoost procedure: the weak leamer is constrained so 
that each weak classifier retumed can depend on only a single feature [17]. As a result 
each stage of the boosting process, which selects a new weak classifier, can be viewed 
as a feature selection process. AdaBoost provides an effective learning algorithm and 
strong bounds on generalization performance [12, 7, 8]. 

The third major contribution of this paper is a method for combining successively 
more complex classifiers in a cascade structure which dramatically increases the speed 
of the detector by focussing attention on promising regions of the image. The notion 
behind focus of attention approaches is that it is often possible to rapidly determine 
where in an image an object might occur [18, 6, 1]. More complex processing is re- 
served only for these promising regions. The key measure of such an approach is the 
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"false negative" rate of the attentional process. It must be the case that all, or almost 
all, object instances are selected by the attentional filter. 

We will describe a process for training an extremely simple and efficient classifier 
which can be used as a "supervised" focus of attention operator The term supervised 
refers to the fact that the attentional operator is trained to detect examples of a particular 
class. In the domain of face detection it is possible to achieve fewer than 1% false 
negatives and 40% false positives using a classifier which can be evaluated in 20 simple 
operations (approximately 60 microprocessor instructions). The effect of this filter is 
to reduce by over one half the number of locations where the final detector must be 
evaluated. 

Those sub-windows which are not rejected by the initial classifier are processed by 
a sequence of classifiers, each slightly more complex than the last. If any classifier 
rejects the sub-window, no further processing is performed. The structure of the cas- 
caded detection process is essentially that of a degenerate decision tree, and as such is 
related to the work of Amit and Geman [1]. 

The complete face detection cascade has 32 classifiers, which total over 80,000 
operations. Nevertheless the cascade structure results in extremely rapid average de- 
tection times. On a difficult dataset, containing 507 faces and 75 million sub-windows, 
faces are detected using an average of 270 microprocessor instructions per sub-window. 
In comparison, this system is about 15 times faster than an implementation of the de- 
tection system constructed by Rowley et al. ^ [1 1] 

An extremely fast face detector will have broad practical applications. These in- 
clude user interfaces, image databases, and teleconferencing. This increase in speed 
will enable real-time face detection applications on systems where they were previ- 
ously infeasible. In applications where rapid frame-rates are not necessary, our system 
will allow for significant additional post-processing and analysis. In addition our sys- 
tem can be implemented on a wide range of small low power devices, including hand- 
helds and embedded processors. In our lab we have implemented this face detector 
on the Compaq iPaq handheld and have achieved detection at two frames per second 
(this device has a low power 200 mips Strong Arm processor which lacks floating point 
hardware). 

1.1 Overview 

The remaining sections of the paper will discuss the implementation of the detector, 
related theory, and experiments. Section 2 will detail the form of the features as well 
as a new scheme for computing them rapidly. Section 3 will discuss the method in 
which these features are combined to form a classifier. The machine learning method 
used, a variant of AdaBoost, also acts as a feature selection mechanism. While the 
classifiers that are constructed in this way have good computational and classification 
performance, they are far too slow for a real-time classifier. Section 4 will describe 
a method for constructing a cascade of classifiers which together yield an extremely 

'Henry Rowley very graciously supplied us with implementations of his detection system for direct com- 
parison. Reported results are against his fastest system. It is difficult to determine from the pubUshed 
literature, but the Rowley-Baluja-Kanade detector is widely considered the fastest detection system and has 
been heavily tested on real-world problems. 
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Figure 1 : Example rectangle features shown relative to the enclosing detection window. 
The sum of the pixels which lie within the white rectangles are subtracted from the 
sum of pixels in the grey rectangles. Two-rectangle features are shown in (A) and (B). 
Figure (C) shows a three-rectangle feature, and (D) a four-rectangle feature. 



reliable and efficient object detector. Section 5 will describe a number of experimen- 
tal results, including a detailed description of our experimental methodology. Finally 
Section 6 contains a discussion of this system and its relationship to related systems. 



2 Features 

Our object detection procedure classifies images based on the value of simple features. 
There are many motivations for using features rather than the pixels directly. The most 
common reason is that features can act to encode ad-hoc domain knowledge that is 
difficult to leam using a finite quantity of training data. For this system there is also a 
second critical motivation for features: the feature-based system operates much faster 
than a pixel-based system. 

The simple features used are reminiscent of Haar basis functions which have been 
used by Papageorgiou et al. [8]. More specifically, we use three kinds of features. The 
value of a two-rectangle feature is the difference between the sum of the pixels within 
two rectangular regions. The regions have the same size and shape and are horizontally 
or vertically adjacent (see Figure 1). A three-rectangle feature computes the sum within 
two outside rectangles subtracted from the sum in a center rectangle. Finally dL four- 
rectangle feature computes the difference between diagonal pairs of rectangles. 

Given that the base resolution of the detector is 24x24, the exhaustive set of rectan- 
gle features is quite large, 45,396 . Note that unlike the Haar basis, the set of rectangle 
features is overcomplete^. 



complete basis has no linear dependence between basis elements and has the same number of elements 
as the image space, in this case 576. The full set of 45,396 thousand features is many times over-complete. 
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Figure 2: The value of the integral image at point (x, y) is the sum of all the pixels 
above and to the left. 



2.1 Integral Image 

Rectangle features can be computed very rapidly using an intermediate representation 
for the image which we call the integral image. ^ The integral image at location x, y 
contains the sum of the pixels above and to the left of x, y, inclusive: 

ii{x,v)= ^ i{x',y'), 

x' <.x,y' <.y 

where ii{x,y) is the integral image and i{x,y) is the original image (see Figure 2). 
Using the following pair of recurrences: 

s{x,y) = s{x,y -1) +i{x,y) (1) 

n{x,y) = n{x-l,y)+s{x,y) (2) 

(where s{x,y) is the cumulative row sum, s{x,—l) = 0, and ii{—l,y) = 0) the 
integral image can be computed in one pass over the original image. 

Using the integral image any rectangular sum can be computed in four array ref- 
erences (see Figure 3). Clearly the difference between two rectangular sums can be 
computed in eight references. Since the two-rectangle features defined above involve 
adjacent rectangular sums they can be computed in six array references, eight in the 
case of the three-rectangle features, and nine for four-rectangle features. 

One alternative motivation for the integral image comes from the "boxlets" work 
of Simard, et al. [15]. The authors point out that in the case of Unear operations (e.g. 



^There is a close relation to "summed area tables" as used in graphics [2]. We choose a different name 
here in order to emphasize its use for the analysis of images, rather than for texture mapping. 
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Figure 3: The sum of the pixels within rectangle D can be computed with four array 
references. The value of the integral image at location 1 is the sum of the pixels in 
rectangle A. The value at location 2 is A + B, at location 3 is ^ + C, and at location 
4is A + B + C + D. The sum within D can be computed as 4 + 1 - (2 + 3). 

/ • g), any invertible linear operation can be applied to / or g if its inverse is applied to 
the result. For example in the case of convolution, if the derivative operator is applied 
both to the image and the kernel the result must then be double integrated: 



The authors go on to show that convolution can be significantly accelerated if the 
derivatives of / and g are sparse (or can be made so). A similar insight is that an 
invertible linear operation can be applied to / if its inverse is appUed to g: 



Viewed in this framework computation of the rectangle sum can be expressed as a 
dot product, i ■ r, where i is the image and r is the box car image (with value 1 within 
the rectangle of interest and 0 outside). This operation can be rewritten 



The integral image is in fact the double integral of the image (first along rows and 
then along columns). The second derivative of the rectangle (first in row and then in 
colunnn) yields four delta functions at the comers of the rectangle. Evaluation of the 
second dot product is accompUshed with four array accesses. 





i ■ r = { 




i)-r". 
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2.2 Feature Discussion 

Rectangle features are somewhat primitive when compared with alternatives such as 
steerable filters [3, 5]. Steerable filters, and their relatives, are excellent for the detailed 
analysis of boundaries, image compression, and texture analysis. In contrast rectangle 
features, while sensitive to the presence of edges, bars, and other simple image struc- 
ture, are quite coarse. Unhke steerable filters the only orientations available are vertical 
and horizontal. It appears as though the set of rectangle features do however provide 
a rich image representation which supports effective learning. The extreme computa- 
tional efficiency of rectangle features provides ample compensation for their limited 
flexibiUty. 

In order to appreciate the computational advantage of the integral image technique, 
consider a more conventional approach in which a pyramid of images is computed. 
Like most object detection systems, our detector scans the input at many scales; start- 
ing at the base scale in which objects are detected at a size of 24x24 pixels, the image 
is scanned at 1 1 scales each a factor of 1.25 larger than the last. The conventional ap- 
proach is to compute a pyramid of 11 images, each 1 .25 times smaller than the previous 
image. A fixed scale detector is then scanned across each of these images. Computa- 
tion of the pyramid, while straightforward, requires significant time. Implemented on 
conventional hardware it is extremely difficult to compute a pyramid at 15 frames per 
second"*. 

In contrast we have defined a meaningful set of features, which have the property 
that a single feature can be evaluated at any scale and location in a few operations. We 
will show in Section 4 that effective face detectors can be constructed with as little as 
two rectangle features. Given the computational efficiency of these features, the face 
detection process can be completed for an entire image at every scale at 15 frames per 
second, less time than is required to evaluate the 1 1 level image pyramid alone. Any 
procedure which requires a pyramid of this type will necessarily run slower than our 
detector. 

3 Learning Classification Functions 

Given a feature set and a training set of positive and negative images, any number of 
machine learning approaches could be used to learn a classification function. Sung and 
Poggio use a mixture of Gaussian model [16]. Rowley, Baluja, and Kanade use a small 
set of simple image features and a neural network [11]. Osuna, et al. used a support 
vector machine [7]. More recently Roth et al. have proposed a new and unusual image 
representation and have used the Winnow learning procedure [10]. 

Recall that there are 45,396 rectangle features associated with each image sub- 
window, a number far larger than the number of pixels. Even though each feature can 
be computed very efficiently, computing the complete set is prohibitively expensive. 
Our hypothesis, which is borne out by experiment, is that a very small number of these 

'^The total number of pixels in the 11 level pyramid is about 55 * 384 * 288 = 6082560. Given that 
each pixel requires 10 operations to compute, the pyramid requires about 60,000,000 operations. About 
900,000,000 operations per second are required to acheive a processing rate of 15 frames per second. 
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features can be combined to form an effective classifier. The main challenge is to find 
these features. 

In our system a variant of AdaBoost is used both to select the features and to train 
the classifier [4]. In its original form, the AdaBoost learning algorithm is used to boost 
the classification performance of a simple learning algorithm (e.g., it might be used 
to boost the performance of a simple perceptron). It does this by combining a collec- 
tion of weak classification functions to form a stronger classifier. In the language of 
boosting the simple learning algorithm is called a weak leamer. So, for example the 
perceptron learning algorithm searches over the set of possible perceptrons and returns 
the perceptron with the lowest classification error. The learner is called weak because 
we do not expect even the best classification function to classify the training data well 
(i.e. for a given problem the best perceptron may only classify the training data cor- 
rectly 51% of the time). In order for the weak learner to be boosted, it is called upon to 
solve a sequence of learning problems. After the first round of learning, the examples 
are re-weighted in order to emphasize those which were incorrectly classified by the 
previous weak classifier. The final strong classifier takes the form of a perceptron, a 
weighted combination of weak classifiers followed by a threshold. ^ 

The formal guarantees provided by the AdaBoost learning procedure are quite 
strong. Freund and Schapire proved that the training error of the strong classifier ap- 
proaches zero exponentially in the number of rounds. More importantly a number of 
results were later proved about generalization performance [13]. The key insight is that 
generalization performance is related to the margin of the examples, and that AdaBoost 
achieves large margins rapidly. 

The conventional AdaBoost procedure can be easily interpreted as a greedy feature 
selection process. Consider the general problem of boosting, in which a large set of 
classification functions are combined using a weighted majority vote. The challenge is 
to associate a large weight with each good classification function and a smaller weight 
with poor functions. AdaBoost is an aggressive mechanism for selecting a small set of 
good classification functions which nevertheless have significant variety. Drawing an 
analogy between weak classifiers and features, AdaBoost is an effective procedure for 
searching out a small number of good "features" which nevertheless have significant 
variety. 

One practical method for completing this analogy is to restrict the weak leamer to 
the set of classification functions each of which depend on a single feature. In support 
of this goal, the weak learning algorithm is designed to select the single rectangle 
feature which best separates the positive and negative examples (this is similar to the 
approach of [17] in the domain of image database retrieval). For each feature, the weak 
learner determines the optimal threshold classification function, such that the minimum 
number of examples are misclassified. A weak classifier {hj{x)) thus consists of a 
feature (fj), a threshold {0j) and a parity (pj) indicating the direction of the inequaUty 
sign: 



'in the case where the weak leamer is a perceptron learning algorithm, the final boosted classifier is 
a two layer perceptron. A two layer perceptron is in principle much more powerful than any single layer 
perceptron. 
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Here a; is a 24x24 pixel sub-window of an image. 

In practice no single feature can perform the classification task with low error. 
Features which are selected early in the process yield error rates between 0.1 and 0.3. 
Features selected in later rounds, as the task becomes more difficult, yield error rates 
between 0.4 and 0.5. Table 1 shows the learning algorithm. 



Given example images (xi,yi), . . . , {xn,yn) where yi = 0,1 for negative and 
positive examples respectively. 



• Initiahze weights wi^, = 57 for = 0, 1 respectively, where m and I are 
the number of negatives and positives respectively. 

• Fori = 1,...,T: 

1. Normalize the weights, 

Wt,i 



En 



so that u't is a probability distribution. 

2. For each feature, j, train a classifier hj which is restricted to using 
a single feature. The error is evaluated with respect to wt, = 

J2iWi \hj{xi) -yi\. 

3. Choose the classifier, ht, with the lowest error e^. 

4. Update the weights: 

where = 0 if example Xi is classified correctly, = 1 otherwise, and 
• The final strong classifier is: 

\ 0 otherwise 

where at = log ^ 



Table 1 : The boosting algorithm for learning a query online. T hypotheses are con- 
structed each using a single feature. The final hypothesis is a weighted linear combi- 
nation of the T hypotheses where the weights are inversely proportional to the training 
errors. 



3.1 Learning Discussion 

Many general feature selection procedures have been proposed (see chapter 8 of [19] 
for a review). Our final application demanded a very aggressive process which would 



3.2 Learning Results 
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discard the vast majority of features. For a similar recognition problem Papageorgiou 
et al. proposed a scheme for feature selection based on feature variance [8]. They 
demonstrated good results selecting 37 features out of a total 1734 features. While this 
is a significant reduction, the number of features evaluated for every image sub- window 
is still reasonably large. 

Roth et al. propose a feature selection process based on the Winnow exponential 
perceptron learning rule [10]. These authors use a very large and unusual feature set, 
where each pixel is mapped into a binary vector of d dimensions (when a particular 
pixel takes on the value x, in the range [0, d — 1], the a;-th dimension is set to 1 and the 
other dimensions to 0). The binary vectors for each pixel are concatenated to form a 
single binary vector with nd dimensions (n is the number of pixels). The classification 
rule is a perceptron, which assigns one weight to each dimension of the input vector. 
The Winnow learning process converges to a solution where many of these weights are 
zero. Nevertheless a very large number of features are retained (perhaps a few hundred 
or thousand). 

3.2 Learning Results 

While details on the training and performance of the final system are presented in Sec- 
tion 5, several simple results merit discussion. Initial experiments demonstrated that a 
classifier constructed from 200 features would yield reasonable results (see Figure 4). 
Given a detection rate of 95% the classifier yielded a false positive rate of 1 in 14084 
on a testing dataset. 

For the task of face detection, the initial rectangle features selected by AdaBoost 
are meaningful and easily interpreted. The first feature selected seems to focus on the 
property that the region of the eyes is often darker than the region of the nose and 
cheeks (see Figure 5). This feature is relatively large in comparison with the detection 
sub-window, and should be somewhat insensitive to size and location of the face. The 
second feature selected relies on the property that the eyes are darker than the bridge 
of the nose. 

In sunnmary the 200-feature classifier provides initial evidence that a boosted clas- 
sifier constructed from rectangle features is an effective technique for object detection. 
In terms of detection, these results are compelling but not sufficient for many real- world 
tasks. In terms of computation, this classifier is probably faster than any other published 
system, requiring 0.7 seconds to scan an 384 by 288 pixel image. Unfortunately, the 
most straightforward technique for improving detection performance, adding features 
to the classifier, directly increases computation time. 

4 The Attentional Cascade 

This section describes an algorithm for constructing a cascade of classifiers which 
achieves increased detection performance while radically reducing computation time. 
The key insight is that smaller, and therefore more efficient, boosted classifiers can be 
constructed which reject many of the negative sub- windows while detecting almost all 
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ROC curve for 200 feature classifier 
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Figure 4: Reciever operating characteristic (ROC) curve for the 200 feature classifier. 




Figure 5: The first and second features selected by AdaBoost. The two features are 
shown in the top row and then overlayed on a typical training face in the bottom row. 
The first feature measures the difference in intensity between the region of the eyes and 
a region across the upper cheeks. The feature capitalizes on the observation that the 
eye region is often darker than the cheeks. The second feature compares the intensities 
in the eye regions to the intensity across the bridge of the nose. 
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positive instances. Simpler classifiers are used to reject the majority of sub-windows 
before more complex classifiers are called upon to achieve low false positive rates. 

Stages in the cascade are constructed by training classifiers using AdaBoost. Start- 
ing with a two-feature strong classifier, an effective face filter can be obtained by ad- 
justing the strong classifier threshold to minimize false negatives. The initial AdaBoost 
threshold, ^ Xlt^i ^t, is designed to yield a low error rate on the training data. A lower 
threshold yields higher detection rates and higher false positive rates. Based on per- 
formance measured using a validation training set, the two-feature classifier can be 
adjusted to detect 100% of the faces with a false positive rate of 40%. See Figure 5 for 
a description of the two features used in this classifier. 

The detection performance of the two-feature classifier is far from acceptable as an 
object detection system. Nevertheless the classifier can significantly reduce the number 
of sub- windows that need further processing with very few operations: 

1. Evaluate the rectangle features (requires between 6 and 9 array references per 
feature). 

2. Compute the weak classifier for each feature (requires one threshold operation 
per feature). 

3. Combine the weak classifiers (requires one multiply per feature, an addition, and 
finally a threshold). 

A two feature classifier amounts to about 60 microprocessor instructions. It seems 
hard to imagine that any simpler filter could achieve higher rejection rates. By compar- 
ison, scanning a simple image template, or a single layer perceptron, would require at 
least 20 times as many operations per sub- window. 

The overall form of the detection process is that of a degenerate decision tree, what 
we call a "cascade" [9] (see Figure 6). A positive result from the first classifier triggers 
the evaluation of a second classifier which has also been adjusted to achieve very high 
detection rates. A positive result from the second classifier triggers a third classifier, 
and so on. A negative outcome at any point leads to the itmnediate rejection of the 
sub-window. 

The structure of the cascade reflects the fact that within any single image an over- 
whelming majority of sub- windows are negative. As such, the cascade attempts to 
reject as many negatives as possible at the earliest stage possible. While a positive 
instance will trigger the evaluation of every classifier in the cascade, this is an exceed- 
ingly rare event. 

Much like a decision tree, subsequent classifiers are trained using those examples 
which pass through all the previous stages. As a result, the second classifier faces a 
more difficult task than the first. The examples which make it through the first stage are 
"harder" than typical examples. The more difficult examples faced by deeper classifiers 
push the entire reciever operating characteristic (ROC) curve downward. At a given 
detection rate, deeper classifiers have correspondingly higher false positive rates. 
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Figure 6: Schematic depiction of a the detection cascade. A series of classifiers are ap- 
plied to every sub-window. The initial classifier eliminates a large number of negative 
examples with very little processing. Subsequent layers eliminate additional negatives 
but require additional computation. After several stages of processing the number of 
sub-windows have been reduced radically. Further processing can take any form such 
as additional stages of the cascade (as in our detection system) or an alternative detec- 
tion system. 



4.1 Training a Cascade of Classifiers 

The cascade design process is driven from a set of detection and performance goals. 
For the face detection task, past systems have achieved good detection rates (between 
85 and 95 percent) and extremely low false positive rates (on the order of 10 ~^ or 
10~^). The number of cascade stages and the size of each stage must be sufficient to 
achieve similar detection performance while minimizing computation. 

Given a trained cascade of classifiers, the false positive rate of the cascade is 

where F is the false positive rate of the cascaded classifier, K is the number of clas- 
sifiers, and fi is the false positive rate of the ith classifier on the examples that get 
through to it. The detection rate is 

K 

D = l[di, 

where D is the detection rate of the cascaded classifier, K is the number of classifiers, 
and di is the detection rate of the ith classifier on the examples that get through to it. 

Given concrete goals for overall false positive and detection rates, target rates can 
be determined for each stage in the cascade process. For example a detection rate of 
0.9 can be achieved by a 10 stage classifier if each stage has a detection rate of 0.99 
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(since 0.9 Ki 0.99^°). While achieving this detection rate may sound like a daunting 
task, it is made significantly easier by the fact that each stage need only achieve a false 
positive rate of about 30% (0.30 1° fti 6 x 10"*'). 

The number of features evaluated when scanning real images is necessarily a prob- 
abiUstic process. Any given sub-window will progress down through the cascade, one 
classifier at a time, until it is decided that the window is negative or, in rare circum- 
stances, the window succeeds in each test and is labelled positive. The expected be- 
havior of this process is determined by the distribution of image windows in a typical 
test set. The key measure of each classifier is its "positive rate", the proportion of win- 
dows which are labelled as potentially containing the object of interest. The expected 
number of features which are evaluated is: 



where N is the expected number of features evaluated, K is the number of classifiers. 
Pi is the positive rate of the ith classifier, and n ; are the number of features in the iih 
classifier. Interestingly, since objects are extremely rare the "positive rate" is effectively 
equal to the false positive rate. 

The process by which each element of the cascade is trained requires some care. 
The AdaBoost learning procedure presented in Section 3 attempts only to minimize 
errors, and is not specifically designed to achieve high detection rates at the expense 
of large false positive rates. One simple, and very conventional, scheme for trading off 
these errors is to adjust the threshold of the perceptron produced by AdaBoost. Higher 
thresholds yield classifiers with fewer false positives and a lower detection rate. Lower 
thresholds yield classifiers with more false positives and a higher detection rate. It is 
not clear, at this point, whether adjusting the threshold in this way preserves the training 
and generalization guarantees provided by AdaBoost. 

The overall training process involves two types of tradeoffs. In most cases clas- 
sifiers with more features will achieve higher detection rates and lower false positive 
rates. At the same time classifiers with more features require more time to compute. In 
principle one could define an optimization framework in which 

• the number of classifier stages, 

• the number of features, n j, of each stage, 

• the threshold of each stage 

are traded off in order to minimize the expected number of features N given a target 
for F and D. Unfortunately finding this optimum is a tremendously difficult problem. 

In practice a very simple framework is used to produce an effective classifier which 
is highly efficient. The user selects the minimum acceptable rates for /j and dj. Each 
layer of the cascade is trained by AdaBoost (as described in Table 1) with the number of 
features used being increased until the target detection and false positves rates are met 
for this level. The rates are determined by testing the current detector on a validation 
set. If the overall target false positive rate is not yet met then another layer is added to 
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the cascade. The negative set for training subsequent layers is obtained by collecting 
all false detections found by running the current detector on a set of images which do 
not contain any instances of the object. This algorithm is given more precisely in Table 
2. 

• User selects values for /, the maximum acceptable false positive rate per layer 
and d, the minimum acceptable detection rate per layer. 

• User selects target overall false positive rate, Ftarget- 

• P = set of positive examples 

• iV = set of negative examples 

• Fo = 1.0; Do = 1.0 

• 1 = 0 

• while Fi > Ftarget 

- i -i^ i + 1 

- ni=0;Fi = Fi_i 

- while Fi> fx Fi_i 

* n, n, + 1 

* Use P and N to train a classifier with rii features using AdaBoost 

* Evaluate current cascaded classifier on vaUdation set to determine F^ 
and Di. 

* Decrease threshold for the ith classifier until the current cascaded clas- 
sifier has a detection rate of at least dx Di-i (this also affects F^) 

- iV ^ 0 

- If Fj > Ftarget then evaluate the current cascaded detector on the set of 
non-face images and put any false dectections into the set N 

Table 2: The training algorithm for building a cascaded detector. 
4.2 Simple Experiment 

In order to explore the feasibility of the cascade approach two simple detectors were 

trained: a monolithic 200-feature classifier and a cascade of ten 20-feature classifiers. 
The first stage classifier in the cascade was trained using 5000 faces and 10000 non- 
face sub-windows randomly chosen from non-face images. The second stage classifier 
was trained on the same 5000 faces plus 5000 false positives of the first classifier This 
process continued so that subsequent stages were trained using the false positives of 
the previous stage. 

The monolithic 200-feature classifier was trained on the union of all examples used 
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to train all the stages of the cascaded classifier. Note that without reference to the 
cascaded classifier, it might be difficult to select a set of non-face training examples to 
train the monolithic classifier. We could of course use all possible sub-windows from 
all of our non-face images, but this would make the training time impractically long. 
The sequential way in which the cascaded classifier is trained effectively reduces the 
non-face training set by throwing out easy examples and focusing on the "hard" ones. 

Figure 7 gives the ROC curves comparing the performance of the two classifiers. 
It shows that there is little difference between the two in terms of accuracy. However, 
there is a big difference in terms of speed. The cascaded classifier is nearly 10 times 
faster since its first stage throws out most non-faces so that they are never evaluated by 
subsequent stage. 
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Figure 7: ROC curves comparing a 200-feature classifier with a cascaded classifier 
containing ten 20-feature classifiers. Accuracy is not significantly different, but the 
speed of the cascaded classifier is almost 10 times faster. 



4.3 Detector Cascade Discussion 

A notion similar to the cascade appears in the face detection system described by Row- 
ley et al. [1 1]. Rowley et al. trained two neural networks. One network was moderately 
complex, focused on a small region of the image, and detected faces with a low false 
positive rate. They also trained a second neural network which was much faster, fo- 
cused on a larger regions of the image, and detected faces with a higher false positive 
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rate. Rowley et al. used the faster second network to prescreen the image in order to 
find candidate regions for the slower more accurate network. Though it is difficult to 
determine exactly, it appears that Rowley et al.'s two network face system is the fastest 
existing face detector'' Our system uses a similar approach, but it extends this two 
stage cascade to include 32 stages. 

The structure of the cascaded detection process is essentially that of a degenerate 
decision tree, and as such is related to the work of Amit and Geman [1]. Unlike tech- 
niques which use a fixed detector, Amit and Geman propose an alternative point of 
view where unusual co-occurrences of simple image features are used to trigger the 
evaluation of a more complex detection process. In this way the full detection process 
need not be evaluated at many of the potential image locations and scales. While this 
basic insight is very valuable, in their implementation it is necessary to first evaluate 
some feature detector at every location. These features are then grouped to find unusual 
co-occiurences. In practice, since the form of our detector and the features that it uses 
are extremely efficient, the amortized cost of evaluating our detector at every scale and 
location is much faster than finding and grouping edges throughout the image. 

5 Results 

This section describes the final face detection system. The discussion includes details 
on the structure and training of the cascaded detector as well as results on a large real- 
world testing set. 

5.1 Training Dataset 

The face training set consisted of 4916 hand labeled faces scaled and aligned to a 
base resolution of 24 by 24 pixels. The faces were extracted from images downloaded 
during a random crawl of the world wide web. Some typical face examples are shown 
in Figure 8. Notice that these examples contain more of the head than the examples 
used by Rowley or et al. [1 1] or Sung [16]. Initial experiments also used 16 by 16 pixel 
training images in which the faces were more tightly cropped, but got slightly worse 
results. Presumably the 24 by 24 examples include extra visual information such as 
the contours of the chin and cheeks and the hair line which help to improve accuracy. 
Because of the nature of the features used, the larger sized sub-windows do not slow 
performance. In fact, the additional information contained in the larger sub- windows 
could be used to reject non- faces earUer in the detection cascade. 

5.2 Structure of the Detector Cascade 

The final detector is a 32 layer cascade of classifiers which includes a total of 4297 
features. 

The first classifier in the cascade is constructed using two features and rejects about 
60% of non-faces while correctly detecting close to 100% of faces. The next classifier 



*Other published detectors have either neglected to discuss performance in detail, or have never published 
detection and false positive rates on a large and difficult training set. 
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Figure 8: Example of frontal upright face images used for training. 



has five features and rejects 80% of non-faces while detecting almost 100% of faces. 
The next three layers are 20-feature classifiers followed by two 50-feature classifiers 
followed by five 100-feature classifiers and then twenty 200-feature classifiers. The 
particular choices of number of features per layer was driven through a trial and error 
process in which the number of features were increased until a significant reduction 
in the false positive rate could be achieved. More levels were added until the false 
positive rate on the validation set was nearly zero while still maintaining a high correct 
detection rate. The final number of layers, and the size of each layer, are not critical to 
the final system performance. 

The two, five and first twenty-feature classifiers were trained with the 4916 faces 
and 10,000 non-face sub- windows (also of size 24 by 24 pixels) using the Adaboost 
training procedure described in Table 1. The non-face sub- windows were collected by 
selecting random sub-windows from a set of 9500 images which did not contain faces. 
Different sets of non-face sub-windows were used for training the different classifiers 
to ensure that they were somewhat independent and didn't use the same features. 

The non-face examples used to train subsequent layers were obtained by scanning 
the partial cascade across large non-face images and collecting false positives. A max- 
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imum of 6000 such non-face sub-windows were collected for each layer There are 
approximately 350 milUon non-face sub-windows contained in the 9500 non-face im- 
ages. 

Training time for the entire 32 layer detector was on the order of weeks on a single 
466 MHz AlphaStation XP900. During this laborious training process several improve- 
ments to the learning algorithm were discovered. These improvements, which will be 
described elsewhere, yield a 100 fold decrease in training time. 

5.3 Speed of the Final Detector 

The speed of the cascaded detector is directly related to the number of features eval- 
uated per scanned sub- window. As discussed in section 4.1, the number of features 
evaluated depends on the images being scanned. Evaluated on the MITh-CMU test set 
[11], an average of 8 features out of a total of 4297 are evaluated per sub-window. This 
is possible because a large majority of sub- windows are rejected by the first or second 
layer in the cascade. On a 700 Mhz Pentium III processor, the face detector can pro- 
cess a 384 by 288 pixel image in about .067 seconds (using a starting scale of 1.25 and 
a step size of 1.5 described below). This is roughly 15 times faster than the Rowley- 
Baluja-Kanade detector [11] and about 600 times faster than the Schneiderman-Kanade 
detector [14]. 

5.4 Image Processing 

AH example sub- windows used for training were variance normalized to minimize the 
effect of different lighting conditions. Normalization is therefore necessary during 
detection as well. The variance of an image sub-window can be computed quickly 
using a pair of integral images. Recall that = m'^ — j^'^x^, where a is the standard 
deviation, m is the mean, and x is the pixel value within the sub-window. The mean 
of a sub-window can be computed using the integral image. The sum of squared pixels 
is computed using an integral image of the image squared (i.e. two integral images are 
used in the scanning process). During scanning the effect of image normalization can 
be achieved by post multiplying the feature values rather than operating on the pixels. 

5.5 Scanning the Detector 

The final detector is scanned across the image at multiple scales and locations. Scaling 
is achieved by scaling the detector itself, rather than scaling the image. This process 
makes sense because the features can be evaluated at any scale with the same cost. 
Good results were obtained using a set of scales a factor of 1.25 apart. 

The detector is also scanned across location. Subsequent locations are obtained by 
shifting the window some number of pixels A. This shifting process is affected by the 
scale of the detector: if the current scale is s the window is shifted by [sA], where [] is 
the rounding operation. 

The choice of A affects both the speed of the detector as well as accuracy. Since the 
training images have some translational variability the learned detector achieves good 
detection performance in spite of small shifts in the image. As a result the detector 
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sub-window can be shifted more than one pixel at a time. However, a step size of more 
than one pixel tends to decrease the detection rate slightly while also decreasing the 
number of false positives. We present results for two different step sizes. 

5.6 Integration of Multiple Detections 

Since the final detector is insensitive to small changes in translation and scale, multiple 
detections will usually occur around each face in a scanned image. The same is often 
true of some types of false positives. In practice it often makes sense to return one final 
detection per face. Toward this end it is useful to postprocess the detected sub-windows 
in order to combine overlapping detections into a single detection. 

In these experiments detections are combined in a very simple fashion. The set 
of detections are first partitioned into disjoint subsets. Two detections are in the same 
subset if their bounding regions overlap. Each partition yields a single final detection. 
The comers of the final bounding region are the average of the comers of all detections 
in the set. 

In some cases this postprocessing decreases the number of false positives since an 
overlapping subset of false positives is reduced to a single detection. 

5.7 Experiments on a Real- World Test Set 

We tested our system on the MITh-CMU frontal face test set [11]. This set consists of 
130 images with 507 labeled frontal faces. A ROC curve showing the performance of 
our detector on this test set is shown in Figure 9. To create the ROC curve the threshold 
of the perceptron on the final layer classifier is adjusted from -|-oo to — oo. Adjusting 
the threshold to -l-oo will yield a detection rate of 0.0 and a false positive rate of 0.0. 
Adjusting the threshold to — oo, however, increases both the detection rate and false 
positive rate, but only to a certain point. Neither rate can be higher than the rate of the 
detection cascade minus the final layer. In effect, a threshold of — oo is equivalent to 
removing that layer. Further increasing the detection and false positive rates requires 
decreasing the threshold of the next classifier in the cascade. Thus, in order to constmct 
a complete ROC curve, classifier layers are removed. We use the number of false 
positives as opposed to the rate of false positives for the x-axis of the ROC curve to 
facilitate comparison with other systems. To compute the false positive rate, simply 
divide by the total number of sub- windows scanned. For the case of A = 1.0 and 
starting scale = 1.0, the number of sub-windows scanned is 75,081,800. For A = 1.5 
and starting scale = 1.25, the number of sub-windows scanned is 1 8,901,947. 

Unfortunately, most previous published results on face detection have only included 
a single operating regime (i.e. single point on the ROC curve). To make comparison 
with our detector easier we have listed our detection rate for the same false positive 
rate reported by the other systems. Table 3 Usts the detection rate for various num- 
bers of false detections for our system as well as other published systems. For the 
Rowley-Baluja-Kanade results [11], a number of different versions of their detector 
were tested yielding a number of different results. While these various results are not 
actually points on a ROC curve for a particular detector, they do indicate a number 
of different performance points that can be achieved with their approach. They did 
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publish ROC curves for two of their detectors, but these ROC curves did not represent 
their best results. Thus the detection rates listed in the table below for the Rowley- 
Baluja-Kanade detector are actually results for different versions of their detector. For 
the Roth-Yang-Ahuja detector [10], they reported their result on the MIT+CMU test 
set minus 5 images containing line drawn faces removed. So their results are for a 
subset of the MIT+CMU test set containing 125 images with 483 faces. Presumably 
their detection rate would be lower if the full test set was used. The parentheses aroimd 
their detection rate indicates this slightly different test set. 
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Table 3: Detection rates for various numbers of false positives on the MITh-CMU test 
set containing 130 images and 507 faces. 



The Sung and Poggio face detector [16] was tested on the MIT subset of the 
MITh-CMU test set since the CMU portion did not exist yet. The MIT test set con- 
tains 23 images with 149 faces. They achieved a detection rate of 79.9% with 5 false 
positives. Our detection rate with 5 false positives is 77.8% on the MIT test set. 

Figure 10 shows the output of our face detector on some test images from the 
MIT-hCMU test set. 

6 Conclusions 

We have presented an approach for object detection which minimizes computation time 
while achieving high detection accuracy. The approach was used to construct a face 
detection system which is approximately 15 faster than any previous approach. Pre- 
liminary experiments, which will be described elsewhere, show that highly efficient 
detectors for other objects, such as pedestrians, can also be constructed in this way. 

This paper brings together new algorithms, representations, and insights which are 
quite generic and may well have broader application in computer vision and image 
processing. 

The first contribution is a new a technique for computing a rich set of image fea- 
tures using the integral image. In order to achieve true scale invariance, almost all 
object detection systems must operate on muUple image scales. The integral image, by 
eliminating the need to compute a multi-scale image pyramid, reduces the initial image 
processing required for object detection significantly. In the domain of face detection 
the advantage is quite dramatic. Using the integral image, face detection is completed 
before an image pyramid can be computed. 
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Figure 9: ROC curves for our face detector on the MIT+CMU test set. The detector 

was run once using a step size of 1.0 and starting scale of 1.0 (75,081,800 sub-windows 
scanned) and then again using a step size of 1.5 and starting scale of 1.25 (18,901,947 
sub-windows scanned). In both cases a scale factor of 1.25 was used. 



While the integral image should also have immediate use for other systems which 
have used Harr-like features (such as Papageorgiou et al. [8]), it can foreseeably have 
impact on any task where Harr-like features may be of value. Initial experiments have 
shown that a similar feature set is also effective for the task of parameter estimation, 
where the expression of a face, the position of a head, or the pose of an object is 
determined. 

The second contribution of this paper is a technique for feature selection based 
on AdaBoost. An aggressive and effective technique for feature selection will have 
impact on a wide variety of learning tasks. Given an effective tool for feature selection, 
the system designer is free to define a very large and very complex set of features as 
input for the learning process. The resulting classifier is nevertheless computationally 
efficient, since only a small number of features need to be evaluated during run time. 
Frequently the resulting classifier is also quite simple; within a large set of complex 
features it is more likely that a few critical features can be found which capture the 
structure of the classification problem in a straightforward fashion. 

The third contribution of this paper is a technique for constructing a cascade of 
classifiers which radically reduce computation time while improving detection accu- 
racy. Early stages of the cascade are designed to reject a majority of the image in order 
to focus subsequent processing on promising regions. One key point is that the cas- 
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cade presented is quite simple and homogeneous in structure. Previous approaches for 
attentive filtering, such as Itti et. al., propose a more complex and heterogeneous mech- 
anism for filtering [6]. Similarly Amit and Geman propose a hierarchical structure for 
detection in which the stages are quite different in structure and processing f l]. A ho- 
mogenous system, besides being easy to implement and understand, has the advantage 
that simple tradeoffs can be made between processing time and detection performance. 

Finally this paper presents a set of detailed experiments on a difficult face detec- 
tion dataset which has been widely studied. This dataset includes faces under a very 
wide range of conditions including: illumination, scale, pose, and camera variation. 
Experiments on such a large and complex dataset are difficult and time consuming. 
Nevertheless systems which work under these conditions are unlikely to be brittle or 
limited to a single set of conditions. More importantly conclusions drawn from this 
dataset are unlikely to be experimental artifacts. 
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Figure 10: Output of our face detector on a number of test images from the MIT+CMU 
test set. 
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